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Introduction 


In  the  SE  community,  it  is  important  for  systems  to  be  agile  and  rapidly  and  effectively  adapt  to 
sudden  changes  in  the  environment.  Agility  in  SE  is  found  in  two  general  areas  -  process  and 
product.  Process  agility  provides  systems  engineers  with  the  methods,  processes  and  tools 
necessary  to  operate  more  effectively  in  development  environments  driven  by  change.  The 
ability  to  rapidly  adapt  is  necessary  while  working  with  an  increasing  rate  of  technology 
advancement,  an  increasing  need  for  interoperability  between  legacy  and  new  capabilities, 
evolving  requirements  throughout  the  development  lifecycle,  and  the  changing  economic  and 
political  factors  that  undergird  and  enable  system  development.  Perhaps  one  of  the  most 
important  concepts  in  Agile  SE  is  the  reconciliation  and  integration  of  systems  and  software 
engineering  activities.  If  software  development  processes  are  to  operate  seamlessly  with  SE 
processes,  SE  processes  must  borrow  notions  of  agility  and  flexibility  found  in  software 
engineering. 

The  purpose  of  RT-124  is  to  identify,  describe,  and  evaluate  possible  methods,  practices  or  tools 
(enablers)  that  could  improve  the  ability  of  systems  engineering  to  adapt  to  changing 
development  environments.  In  order  to  efficiently  make  use  of  scarce  research  resources,  RT- 
124  has  established  a  triage  process  for  identifying  and  then  rapidly  evaluating  the  probability  of 
effectiveness  of  candidate  enablers  as  they  are  identified.  The  ultimate  result  of  the  process  is  an 
evaluation  white  paper  supporting  one  of  three  decisions: 

1.  not  likely  to  be  effective, 

2.  possibly  suitable  but  more  research  is  needed,  or 

3.  definitely  suitable  and  expedited  transition  is  recommended. 

This  paper  describes  the  process  and  its  products.  After  each  execution  of  the  process,  a 
reflection  activity  will  be  held  to  identify  strengths  and  weaknesses  of  the  process  and  to  identify 
and  make  appropriate  improvements. 
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Selection  and  Evaluation  Process  Overview 


The  overall  process,  as  illustrated  in  Figure  1,  leverages  nearly  a  decade  of  research  into  practice 
description,  evaluation  and  dissemination  represented  by  the  DoD  Acquisition  Best  Practices 
Clearinghouse  (BPCh)..l  [1,  2,  3]  The  process  itself  can  operate  concurrently  for  a  number  of 
enablers,  and  the  actual  cadence  can  be  adjusted  by  the  number  of  enablers  under  consideration 
and  the  number  and  availability  of  evaluators. 


Identification 

Enablers  can  be  found  in  many  environments,  disciplines,  and  activities.  Real  value  can  be 
achieved  when  a  process  used  in  one  discipline  can  be  adapted  quickly  to  provide  value  in  a 
different  discipline.  RT-124  attempts  to  identify  enablers  by  monitoring  the  agile,  lean,  and 
adaptive  research  and  practice  ecosystems.  Generally,  the  most  efficient  way  of  tapping  into  the 
communities  is  via  existing  communities  of  practice.  This  can  be  achieved  through  monitoring 
communications  in  social  media  groups  and  websites  (such  as  Linkedln  or  Facebook  groups 
associated  with  the  Scaled  Agile  Framework,  Lean  Enterprise  Institute,  Agile  Alliance,  Lean 
Systems  Society  and  Model-based  Systems  Engineering),  reading  conference  proceedings, 
attending  workshops,  and  participating  in  working  groups  (such  as  the  INCOSE  Agile  Systems 
Engineering  WG). 

Identification,  however  needs  to  employ  a  set  of  common  criteria  so  that  obviously  inappropriate 
enablers  are  not  pursued.  The  identification  criteria  developed  for  RT-124  are  based  on  earlier 
SERC  work.  [4,  5,  6]: 

•  Supports  some  aspect  of  agility  or  leanness  (e.g.  small  batch  size,  incremental/iterative 
development,  value  to  the  customer) 

•  Is  reasonably  defined  (there  is  a  somewhat  standard  definition) 

•  Aligns  with  at  least  one  of  the  SEBOK  systems  engineering  primary  discipline  areas 

•  Sufficient  information  exists  to  characterize  it 


1  Operated  by  DAU,  the  BPCh  was  a  web-enabled  best  practice  repository  and  selection  tool  residing  within  the  DAU  knowledge 
management  system  and  associated  with  DAU's  acquisition  communities  of  practice.  The  BPCh  operated  through  2010. 


Contract  Number:  HQ0034-13-D-00M 


Report  No.  SERC -2015-TO- 049-1 


Task  Order  024,  RT124 


Figure  1:  Overall  Process 


Characterization  and  Evaluation 

Characterization  consists  of  researching  the  identified  enabler,  gathering  any  evidence  about  its 
use  and  the  results,  if  possible  interviewing  organizations  that  have  applied  it,  and  ideally  (but 
rarely),  finding  any  empirical  studies  regarding  it. 

To  provide  for  a  common  language  (ontology)  and  to  enable  continuous  and  consistent 
assimilation  of  information  over  time,  it  is  appropriate  to  establish  attributes  to  describe  each 
enabler.  The  attributes  are  organized  to  support  the  evaluation  criteria.  Characterization 
attributes  and  their  assessment  scale  are  shown  in  Table  1.  The  attributes  are  intentionally  broad 
to  support  a  fairly  rapid  assessment  of  potential.  The  evaluation  criteria  are  shown  in  Table  2. 

Evaluation  activities  are  centered  around  a  single  researcher  identifying  evidence  from  various 
sources,  discussing  the  enabler  with  experts  in  its  creation  or  use  as  well  as  with  system 
engineering  practitioners  and  managers.  This  information  is  then  reflected  in  the  attributes. 
Information  from  the  attribute  evaluation  is  provided  to  the  research  team,  including  a 
statistically  based  score  for  each  criterion.  This  score  is  considered,  but  is  not  the  only  input  to 
the  decision  making  process.  In  general,  the  score  for  impact  and  relevance  take  precedence, 
since  research  can  usually  mitigate  weaknesses  in  maturity  and  adoptability.  However,  lower 
scores  indicate  that  the  team  should  be  very  clear  about  the  relationship  between  the  possible 
benefit  and  the  cost  of  proceeding. 

If  the  researcher  and  the  team  believe  that  the  enabler  is  simply  not  suitable,  or  that  while  it 
may  show  promise,  the  expense  or  extent  of  additional  research  does  not  seem  to  match  the 
benefit,  the  enabler  is  discarded  and  the  information  filed  as  notes.  If  the  team  believes  there  is 
sufficient  merit  to  do  additional  research,  or  if  there  is  an  indication  that  the  enabler  is  already 
applicable,  a  white  paper  is  generated  and  delivered  to  the  sponsor. 
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Table  1:  Enabler  Attributes 


Attribute 

Description 

Agile/Adaptive  Impact  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  Currently  cannot  support  this  attribute  ( score: -1) 

Partial  Support  -  Does  not  negate  this  attribute  (  score:  0) 

Explicit  Support  -  Designed  to  support  this  attribute  (  score:  +1) 

Batch  Size 

Limiting  or  supporting  smaller  batch  sizes  for  SE  activities 

Iteration 

Supporting  iterative  development  capability 

SE  Activity  Value 

Determining  the  value  of  SE  activities  to  support  better  SE  efficiency  and  effectiveness 

Customer  Value 

Accelerating  the  delivery  of  value  to  the  customer 

Work  In  Progress 

Visibility  of  existing  WIP  or  limiting  WIP  to  increase  flow  and  protect  scarce  resources 

Scheduling 

Flexibility  to  handle  multiple  priority  tasks  without  unnecessary  perturbation  of  engineering  flow 

Requirement 

Evolution 

Changing/emergent  requirements  and  the  ability  to  evolve  systems  over  time 

Discipline 

integration 

Better/faster/more  effective  communication  and  more  rapid  integration  between  various 
disciplines  as  changes  occur 

Artifacts 

Development  of  fewer,  higher-value  artifacts  that  are  easier  to  maintain  congruent 

Stakeholder 

Management 

Effective  and  adaptive  balancing  of  stakeholder  needs 

Relevance  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  Currently  cannot  support  this  attribute  (score:  -1) 

Partial  Support  -  Does  not  negate  this  attribute  (  score:  0) 

Explicit  Support  -  Designed  to  support  the  attribute  ( score:  + 1 ) 

Scalability 

Can  apply  to  all  types  of  systems  from  simple  to  ultra-large  SoSs  with  deep  supplier  chains  and 
multiple  concurrent  and  interacting  initiatives. 

Criticality 

Can  apply  where  there  are  stringent  safety,  security,  or  mission-critical  requirements 

Adaptability 

Can  adapt  or  extend  to  apply  to  different  SE  disciplines,  domains  or  development  circumstances 

Maturity  and  Repeatability  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  does  not  currently  meet  this  attribute  ( score:  -1) 

Partial  Support  -  Weakly  meets  this  attribute  (  score:  0) 

Explicit  Support  -  Strongly  meets  the  attribute]  (  score:  +1) 

Definition 

Is  defined  sufficiently  to  be  studies/replicated. 

Experience 

Is  implemented  or  used  in  multiple  instances 

Breadth 

Has  been  applied  over  a  range  of  different  types  of  organizations  or  application  areas  (e.g. 
acquirers,  developers,  integrators;  business,  communications,  defense,  medicine,  space,  cyber¬ 
physical) 

Media  Presence 

Is  meaningfully  referenced  (e.g.  reviews,  analyses,  case  studies)  directly  or  in  analogy  in  technical 
media  (e.g.  journals,  technical  reports,  respected  blogs) 
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Attribute 


Description 


Adoptability  Attributes 

Evaluation  Scale: 

Unknown  -  Not  determinable  at  this  time  (score:  <null>) 

None  -  does  not  currently  meet  this  attribute  (  score:  -1) 

Partial  Support  -  Weakly  meets  this  attribute  (  score:  0) 

Explicit  Support  -  Strongly  meets  the  attribute]  ( score:  +1) 

Ease  of  Use 

Can  be  learned  and  applied  by  non-experts 

Latency 

Impacts  SE  agility  within  an  acceptable  time  frame 

Cost  to  Deploy 

Investment  costs  (e.g.,  special  equipment,  training)  to  implement  the  enabler  are  acceptable 

Cost  to  Use 

Execution  costs  (licenses,  additional  staff  time)  for  the  enabler  are  acceptable 

Table  2:  Evaluation  Criteria 


Criteria 

Description 

Impact 

High  impact  in  at  least  one  agile  attribute  and  some  impact  in  more  than  one 
additional  area 

Relevance 

And 

Maturity  and 

Sufficiently  well  defined  that  implementation  is  portable  to  other  projects;  Used 

Repeatability 

successfully  in  at  least  one  SE-like  context. 

Adoptability 

Are  sufficiently  related  to  the  culture  and  processes  of  current  systems 
engineering  practice  so  as  not  to  be  rejected  by  the  majority  of  the  workforce; 
do  not  require  overly  burdensome  restructuring  of  organizational  governance 
or  statutory  changes 
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Development  of  the  evaluation  white  paper 


Each  white  paper  will  provide  the  following  information: 

Summary  of  Evaluation  Assessment  and  Recommendations  for  the  Enabler 
Part  I:  Description  of  the  Enabler 

A  description  of  the  enabler  including  any  pertinent  information  as  to  its  source,  its  use,  and 
its  relationship  to  other  enablers  or  existing  processes.  This  section  may  be  very  short  or 
significant  depending  on  the  recommendations 

Part  II:  Evaluation  Attributes  and  Assessment 

A  completed  matrix  of  the  attributes  and  assessed  values  (as  defined  in  Table  1),  including 
the  rationale  for  each  assessment  and  a  general  description  of  how  the  enabler  could  be  of 
value  in  improving  the  agility/adaptability/responsiveness  of  systems  engineering,  and  the 
rationale  for  the  decision 

Part  III:  Recommendation  Details 

If  the  recommendation  is  for  further  research,  then  one  or  two  specific 
studies/experiments/analyses  that  would  lead  to  the  enabler's  validation  or  support  its 
transition  should  be  described.  If  the  recommendation  is  for  expedited  transition,  a 
description  of  why  the  team  believes  this  is  possible,  what  type  of  transition  materials  exist 
or  need  to  be  created,  and  identification  of  organizations  that  would  be  appropriate  as  pilots. 
If  the  enabler  is  deemed  not  suitable,  no  further  information  is  required. 

Part  IV:  Previous  Research 

Previous  research  and  experience  in  the  area  of  interest  that  supports  this  possible 
usefulness. 
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